The Distributed Data Service (DDS) architecture provides a framework for highly distributed, hierarchical, multi-protocol caching. DDS is a distributed caching layer that spans an enterprise's network, encompassing multiple LANs interconnected with WAN links. The caching layer's constituent parts are DDS modules installed on file servers, client workstations, and intermediate nodes (routers, switches, and computers). DDS employs TCP/IP for inter-site communications and may therefore be incrementally deployed. Non-DDS nodes appear as “just part of the wire”.
Conceptually, the DDS caching layer slices through each DDS configured computer system at the vnode interface layer. File systems (UFS, VxFS, NTFS, EXT4) and other devices such as video sources and shared memory plug into the bottom of the caching layer and provide permanent file storage or a data sourcing/sinking capability. Client systems plug into the top of the caching layer to access “local” data. Distributed throughout the network, intermediate DDS nodes (routers, switches, and other computers) provide increased scalability and faster file access.
The DDS layer implements an intelligent integrated data streaming and caching mechanism to make file data appear as “local” as possible. When a client process accesses a file, the file appears to be local (in terms of file access performance) if it has been accessed before and has not been modified since its last access. When file data must be fetched from the origin server, DDS pre-fetches file data in advance of the client's request stream. Of course, pre-fetching is only performed for well-behaved clients. Write behind is also implemented by DDS in a manner consistent with the fact that users aren't very tolerant of file systems that lose their data.
Data cached within the DDS layer is stored in a protocol neutral format in a manner that requires no “translation” when the client is of the same type (Unix, Windows, . . . ) as the origin server.
The DDS layer maintains “UFS consistency” (a read always returns the most recently written data) on cached images and provides several methods of handling and recovering from network partitioning events. To the maximum extent possible, recovery and reconnection is performed automatically with no requirement for user or administrator intervention.
This document discloses and explains the methods and procedures employed by DDS to transparently overcome network infrastructure failures. In this context, “transparent” means that when an intermediate network node or link fails during a DDS file access operation, an alternate path to the origin server is discovered and used to complete the operation without the client or server ever even becoming aware of the network failure.